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(54) Temporal-spatial error concealment apparatus and method for video signal processors 



(57) A video decompression system includes block 
error concealment means (22,41,42) tor generating 
substitute data for lost blocks of pixel data. An interpola- 
tor generates both spatially (44) and temporally (40) 
interpolated or predicted blocks of data for concealing a 
lost block. Pixel data in the respective blocks of spatially 
and interpolated data are transformed (43) to coeffi- 



cients representing frequency spectra. A substitute 
bock of coefficients is assembled from coefficients from 
both the transformed blocks, according to a predeter- 
mined criteria The substitute block is transformed back 
to the spatial domain for substitution of lost pixel values. 
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FIGURE 9 is a block diagram of apparatus for generating combined spatio-temporal interpolated error concealment 
blocks. 

FIGURE 10 is a block diagram of apparatus for calculating a variable parameter "e" used in the FIGURE 9 appara- 
tus. 

s In order to perform block based error concealment it is necessary to be able to determine lost or damaged blocks 

of signal. To this end it is assumed that the transmitted signal to be processed in the exemplary embodiments is format- 
ted in transport packets. Each packet includes a header, a payload, and continuity and/or check bytes. The header 
includes sufficient information to identify the spatial locations, in the reproduced image, of the respective blocks of com- 
pressed video data carried in the payload. The check bytes may be of a form to permit error detection of substantially 

10 all potential errors and at least partial error correction. An example of this type of signal format may be found in United 
states Patent No. 5,247,363. Of course there are other known methods for detecting lost blocks which may also be 
implemented. 

Refer to FIGURE 2 which illustrates a general form of a receiver for processing the above described signals. Trans- 
mitted signals, from for example, an antenna (not shown) are applied to a tuner/demodulator 10. which provides base- 

75 band signal. The baseband signal is applied to a forward error correction, FEC, decoder 12, which examines the 
transmitted signal and corrects errors incurred in the transmission channel, according to its error correction capability. 
Error corrected data from the FEC decoder 12 is applied to a rate buffer 14 which accepts data at a relatively constant 
rate from the FEC decoder 1 2 and outputs data on demand to the subsequent processing elements. Data from the 
buffer is coupled to a frame check sequence. FCS. decoder 1 6, which examines the error corrected data for uncorrected 

20 errors according to FCS check bits (or continuity check count values, for example) appended to the transport packets. 

The FCS decoder 16 passes transport packets, D, to element 18 along with an error signal, ED, indicating whether 
respective transport packets contain data errors. Element 18, responsive to the error data, discards transport packets 
which include data errors. Error free transport packets are separated into video data and transport header data. The 
transport header data and error data, ED, are utilized to determine what data is lost. For more specific information 

25 regarding these processes the reader is invited to examine U.S. Patent application (RCA 86409 A) Sn. 08/01 7,455. filed 
February 12. 1993. and entitled "APPARATUS FOR CONCEALING ERRORS IN A DIGITAL VIDEO PROCESSING 
SYSTEM". Element 18 passes the error free video data and the error tokens ED to the decompressor 20. The error 
tokens may provide several functions. For example, the particular error token may condition the decompressor to simply 
skip over lost blocks in particular frame types. In this instance, decompressed data from the prior frame is simply 

30 repeated from the collocated image area, i.e., a form of temporal interpolation or block replacement. Alternatively, other 
error tokens are passed to a further error concealment processor 26. 

The decompressor 20 utilizes frames of memory 22 in the decompression process. This memory may be arranged 
to store frames of both compressed and decoded video signal for use in the process of motion compensated predictive 
decoding, and error concealment Decoded image information is subsequently transferred to a display memory 24 from 

35 which it may be accessed in raster format for display. The decompressor in this arrangement is conditioned to store 
decoded motion vectors, for at least a portion of a frame, in the memory 22 for use in error concealment. 

The error concealment element 26 interacts with the data in memory 22. and generates substitute image data for 
blocks of pixel values identified by the error tokens. Luminance and chrominance signal components are decompressed 
separately but by similar processes and then recombined for display. The following discussion regarding error conceal - 

40 merrt is directed to concealment of errors in the luminance component Errors in the chrominance components may be 
concealed in similar fashion by similar parallel apparatus not shown. However, parameter evaluation may be performed 
on a single signal component (e.g. luminance) and the results of the evaluation used for determining the type of block 
error concealment for all components. 

FIGURE 3 shows pictorially the functions of one embodiment of the error concealment apparatus 26 of FIGURE 2. 

45 In FIGURE 3. blocks 30 and 31 represent memory from which decompressed data may be accessed. The data from 
element 30 is data which surrounds the lost block B. The data, TR, from element 31 , preferably is a temporal prediction 
of the lost block using motion vectors from the block vertically above the lost block to determine the predicted values. 
Alternatively, block TR may represent data from a prior frame collocated with block B. It should be realized that the 
blocks to the right and left of block B may also be lost or missing if they were contained in the same transport block. 

so However, even if this were the case, the block to the left of block B may be a replacement block if the system performs 
error concealment from left to right, in which case only the block to the right of block B will also be missing. The data 
from elements 30 and 31 are evaluated to determine the mode of error concealment to be executed for the missing 
block B. In order to facilitate this evaluation, it may be beneficial to first replace the missing data of block B and possibly 
the blocks to the right and left of block B (if they too are missing) with some predetermined value, representing gray for 

55 example, or possibly with the average brightness level of the block TR. 

Two or three types of imag evaluation are executed 32, to determine the concealment mode to be performed. A 
first evaluation is for image motion. An example of image motion evaluation, to be described below, looks at the magni- 
tude and relative correlation of the motion vectors for the blocks above and below the missing block. A second evalua- 
tion is of dominant image features such as edges or gradients. An example of edge evaluation, to be described below. 
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The magnitude, G, and angular direction. <I> of the gradient at (ij) are given by: 

G =A/9x 2 +9y 2 <t> = tan- 1 (g y /g x ) (3) 

10 These gradient measures are computed for each (i,j) coordinate in the neighborhood surrounding the missing or dam- 
aged block. The value of the respective gradient angles are rounded to the nearest 22.5 degrees, for example, and thus 
corresponds to one of eight directional categories D1 - D8. If a line drawn through a respective (i,j) neighborhood pixel 
coordinate with the direction determined by the gradient angle will pass through the missing block, that particular (i.j) 
pixel coordinate is given a vote. The directional categories D1 - D8 of the voting coordinates are mutually exclusively 

is accumulated by summing the magnitudes of the associated directional gradients. That is, the magnitudes of the voting 
coordinates having directional category D1 are accumulated to form one sum, the magnitudes of the voting coordinates 
having directional category D2 are accumulated to form a second sum and so on. The directional categories exhibiting 
the largest accumulated sums determine pixel interpolation directions. 

An alternative method of detecting gradient information will be described with reference to FIGURE 5. The circles 

20 in FIGURE 5 represent FFT coefficient values for a superblock. arranged in relation to a vertical frequency axis. f v and 
a horizontal frequency axis, f H . The coefficients represent spatial frequency coefficients for an image and in this exam- 
ple correspond to a 16 by 1 6 pixel matrix. The coefficient at the intersection of the f v and f H axes corresponds to DC or 
the average brightness level over the matrix. Coefficients to the right of the vertical axis f v represent increasing horizon- 
tal frequencies and coefficients above the horizontal axis f H represent increasing vertical frequencies. Coefficients to 

25 the left of the vertical axis and below the horizontal axis represent the conjugates of the foregoing frequency compo- 
nents. 

In FIGURE 5 exemplary coefficient groupings designated 135° FILTER. 90° FILTER and H FILTER are shown. The 
coefficients in the group designated H FILTER include selected coefficients which correspond generally to horizontal 
image gradients. The coefficients in the group designated 90° FILTER include selected coefficients which correspond 

30 generally to vertical image gradients. Trie coefficients in the group designated 135° FILTER include selected coeffi- 
cients which correspond generally to image gradients at 135 degrees to the horizontal etc. Therefore, it may be seen 
that respective groupings can be arranged for a plurality of image gradient directions. To determine the dominant image 
gradient direction, one need only sum the coefficients in the respective groupings (each of which represents a particular 
direction), normalize the sums and compare the respective normalized sums to determine the largest normalized sum. 

35 The angle Theta(M) associated with the group having the largest normalized sum is the dominant image gradient 

Gradient analysis may be performed on the superblock by applying the appropriate pixel data from memory 41 to 
the transform apparatus 43 to generate the corresponding coefficients which will be stored in the memory 41 for the 
above described analysis, which may be performed in the digital signal processor (DSP) 42. The image gracfients of the 
predicted block TR may be similarly determined by transforming the pixel data of block TR in element 43. storing the 

40 coefficients in memory 41 and analyzing respective groupings of coefficients. If this method of gradient analysis is uti- 
lized, the transform coefficients will be retained in memory for further processing described below. 

Image motion analysis is also performed. An exemplary motion analysis generates six motion vector correlation 
measures and forms the weighted average of the six as a directional magnitude correlation measure r m . The motion 
vectors utilized in the analysis are the vectors associated with the blocks designated TOP-1 , TOP, TOP+1 , BOT-1 . BOT. 

45 and BOT+1, located above and below the missing block as shown in FIGURE 6. Note, for MPEG data, motion vectors 
are associated with macroblocks which include a matrix of 2 by 2 luminance representative blocks each representing a 
matrix of 8 by 8 pixels. If block replacement (concealment) is performed on 8 by 8 pixel units, one directional magnitude 
correlation will serve four blocks. However, since in the MPEG environment it is unlikely that only one block of a mac- 
roblock will be lost, but rather that all blocks of respective macroblocks will be concurrently lost, all analysis will be made 

so on a macroblock basis and concealment will be performed on a macroblock basis. It should be assumed that for an 
MPEG environment that the blocks drawn in FIGURE 6 represent macroblocks. 

The motion vectors associated with the blocks neighboring the lost block are accessed from the memory 41 and 
applied to the DSP 42. The DSP 42 is conditioned by the controller 40 to calculate the following correlation factors: 

55 _ V TOP* V TOP-1 

1 = |V T opIIVtop-iI 
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where V (j) is the motion vector associated with block i ForexamnleV ^ . 

r„ -(K1>M»x(r 5 . 0) . (K2)Max<Max(, ,. 0) . (K3)Max(Max(r,, ,.,). 0) ♦ <K4)r 6 . 
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responding vector correlation is determined, a check is made {90} to determine if all desired pairs of blocks (or 
macroblocks) have been processed. If they have not, the next pair is accessed {80} and processed. If they have all been 
processed a correlation measure r m is generated according to the equation ; 



5 n 

r m = £K(n)v(n) 
i 

where K(n) for forward and backward types are the same value, but K(n) for a v(n) associated with a pair of blocks con- 
io taining both forward and backward vectors is larger. 

The temporal correlation measure or error analysis, r p . is performed according to the equation; 

15 15 15 15 

(IX TOPyTOPPij) ^TOPij ^TOPPii 

j=o i=0 i=0 i=0 

1 6 " 16 16 



r P STD ( TOP ) STD ( TOPP ) 



20 



where TOPjj refers to the pixel values in the block above the missing block assuming a block of 16 by 16 pixels. TOPP* 
refers to the pixel values for a predicted block corresponding to the block above the missing block, and STD stands for 
standard deviation. In determining the predicted block TOPP, the motion vectors from block TOP are used unless the 

25 block TOP is an I type block, in which case a zero valued motion vector is used. The calculation for r p may be performed 
in either the DSP 42, the controller 40. or a combination of both. 

The same calculation may be made for the lower block BOT and its temporal prediction BOTP or only the first cal- 
culation may be performed as desired. If a calculation is performed for both the top (TOP) and bottom (BOT) blocks the 
larger of the r p values is selected as the temporal correlation measure r p . 

30 The different correlation measures r p r m , and r d are added to generate a general correlation parameter r. according 
to the relation; 



r=pi(r p ) + p2(r d ) + p3(r m ) 

35 where pi . p2, and p3 are constants having exemplary values of 0.6. 0.2 and 0.2 respectively. Error concealment mode 
is determined by the relative magnitude of r. If r is greater than a threshold TH1 . indicating a high degree of neighboring 
image correlation, concealment of the missing block is by temporal replacement using a predicted block from a prior 
frame. If r is less than a threshold TH2. (TH2 < TH1) indicating relatively low neighboring image correlation, conceal- 
ment of the missing block is by substituting a block generated by spatial interpolation. If TH2 ^ r =s TH1 concealment is 

40 by a combination of spatial interpolation and temporal replacement to be discussed below. Exemplary values of TH1 
and TH2 are 0.9 and 0.4 respectively. 

It is not necessary to calculate and utilize all three of the correlation measures and acceptable results may still be 
achieved using for example only two of the correlation measures. Alternatively other correlation values/measures may 
be added to the mix forming the value r. In addition one or the other of the coefficients pi . p2 and p3 may be set to zero. 

45 The foregoing processes are reviewed in the flowchart of FIGURE 7. It will be appreciated that the order of deter- 
mining the coefficient measures r m , rp, and r d is interchangeable. In FIGURE 7 the measure r p is first calculated {60}. 
Then the appropriate motion vectors are accessed and the correlation measure r m is determined {61 }. The pixel values 
for blocks neighboring a lost block are accessed {62} and the edge gradients r d are determined {63}. The respective 
measures r p . r m and r d are weighted and summed to generate the correlation measure of the spatially and temporally 

so surrounding blocks {64}. 

The value of the correlation parameter r is compared to two threshold values TH1 and TH2 in the concealment con- 
troller 40. If r is greater than TH1 a substitute block for the bad or missing block is generated by temporal replacement. 
This is performed in the controller 40, by substituting the block TR for the missing block in the display portion of memory 
22. for the current frame period. If the value r is less than TH2, then concealment is by substituting a spatially irrterpo- 

55 lated block. This may be performed by accessing the superbiock from the working memory 41 . and applying this pixel 
matrix to the spatial interpolator 44. The interpolator 44 may generate a substitute block by known interpolation meth- 
ods, including directional or bi-directional spatial interpolation in accordance with the directions of the dominant image 
gradient or gradients. Alternatively, if TH2 s r ss TH1, concealment is accomplished with a combination of spatial and 
temporal interpolation, which will be described with reference to FIGURE 9. 



7 



EP0 727 910 A2 



10 



15 



25 



30 



35 



40 



45 



SO 



55 



or g'adSir^Tr^SSVr: ^'^P^^iscussedfcx generating the dominant edges 
dirSonoT^^ gradients. Indicia indicating L 

. . . wyc5) dre ^PP"©" io a gradient mask 904. The superb ock s aoolied tn an fft qoq *k« 

to zero values. This matrix is «J2S»^ ^^t. 'U ^"e rearairmg aoeniciems set 

frame is accessed from memory 912 and acDlied to th* fpt q-m / 1 elw,l,on Ine P reaicte a b,ock TR from a prior 

(1002) are applied to a subtracter 1003 which generates a matrix of difference values ™^hm2™ ^ 
e^C^Zer^XX^ 

e = 6 times cr 

Another value for F(DIFF) = e may be the weighted value of the mean. m. of the coefficients. 

e = K4 times m 



8 



EP0 727 910 A2 

gradient to form a matrix Y[i,fl by selecting from block M', only the M[i,j] coefficients corresponding to those representing 
the dominant gradient, and selecting coefficients from block TR* to provide the remaining coefficient values. 

Claims 

1 . Apparatus for generating substitute blocks of pixel data for missing or corrupted blocks of pixel data, characterized 
by: 

a source (20) of pixel data representing respective images and a source of error data indicating blocks of 
said pixel data that may be missing or corrupted (hereinafter designated bad pixel data); 

a controller (40), responsive to said error data, for accessing from said source, pixel data located spatially 
adjacent to a block of bad pixel data, and a temporally predicted block of data corresponding to said block of bad 
pixel data; 

an interpolator (44) operative upon said pixel data located spatially adjacent to a block of bad pixel data, for 
generating a spatially interpolated block of pixel data; 

transform circuitry (43) for providing a matrix of transform coefficients representing said spatially interpo- 
lated block of pixel data, and for generating a further matrix of transform coefficients representing said temporally 
predicted block of data; 

a selector (926). responsive to a predetermined criteria, for forming a still further matrix of transform coeffi- 
cients, by selecting respective coefficients in said still further matrix exclusively from said matrix of transform coef- 
ficients representing said spatially interpolated block, or from said further matrix of transform coefficients in 
accordance with said criteria; and 

inverse transform circuitry (928) for inverse transforming said still further matrix to generate said substitute 
block of pixel data. 

2. The apparatus set forth in claim 1 characterized in that said interpolator includes digital signal processing appara- 
tus operative upon said pixel data located spatially adjacent to a block of bad pixel data for detecting a direction of 
a dominant image gradient within an image area represented by pixel data located spatially adjacent to a block of 
bad pixel data. 

30 3. The apparatus set forth in daim 2 characterized in that said said selector for forming a still further matrix, generates 
a matrix of transform coefficients including transform coefficients which represent directional image information 
corresponding to said dominant image gradient from said matrix of transform coefficients representing said spa- 
tially interpolated block, and selects transform coefficients from said further matrix of coefficients exclusive of coef- 
ficients which represent directional information corresponding to said dominant edge gradient 

35 

4. The apparatus set forth in claim 1 characterized in that said said selector comprises; 

circuitry for determining respective differences between corresponding transform coefficients in said matrix 
of transform coefficients representing said spatially interpolated block of pixel data, and in said further matrix of 
transform coefficients; 

40 a comparator for comparing respective differences to a predetermined variable "e"; and 

wherein said selector selects a corresponding coefficient from said matrix of transform coefficients repre- 
senting said spatially interpolated block of pixel data, when the corresponding difference is less than said variable 
"e". and selects a corresponding coeff idem from said further matrix of transform coeff idents when the correspond- 
ing difference is greater than the variable "e". 

45 

5. The apparatus set forth in daim 4 characterized in that said said selector further includes means for generating said 
variable n e n from image information located adjacent said bad data. 

6. The apparatus set forth in claim 4 further characterized by: 

50 apparatus responsive to data representing image information of image areas adjacent said block of bad 

pixel data, for determining a measure, r, of correlation of image features in said image areas adjacent said block of 
bad pixel data; 

a source of threshold value TH1 ; 

a comparator for comparing said correlation measure r against the threshold value TH1 and conditioning 
55 said selector t select coeff idents nly from said further matrix if the correlation measure, r, is greater than TH1 , 
and conditioning said selector to select coefficients according to said predetermined criteria if the correlation meas- 
ure, r is less than TH1. 
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The apparatus set forth in claim 6 further characterized by: 

a source of a further threshold value TH2 where TH2 is less than TH1 ■ and 
„ a . ™ h / r * n . oomparator is arranged to compare said conelation measure, r. against said further threshold 
value TH2, and to condrt.cn said selector to select coefficients only from said matrix of transform coSfcSTrSr 

To^^T^'T^ b,0cko, P- 1 data ■ conJL measure, r . i tSSSSSSS^ 
.ng said selector to select coefhcents according to said predetermined criteria if TH2 < r < TH1 cona,,,on 

The apparatus set forth in claim 1 further characterized by; 

mvai JSHX I responsive to date resenting image information of image areas adjacent said block of bad 

C t£££ 09 a measure - r> - corre,afon of image ,eatures h ^ ^ areas ad i acent «S£S 

a source of threshold value THl ; 
means for producing a block of substitute pixel data; 
. . 3 COm P arat ° r for ^"Paring said correlation measure r against the threshold value TH1 and conditionina 
Pr0dUC ' n9 3 U °* * SUbsWtute Pixel to P rovide «" Predicted Wc^cSa ^e 

"r^ 

9. The apparatus set forth in claim 8 further characterized by; 

a source of a threshold value TH2 where TH2 is less than THl • and 
oih ^.^tu^ COm P arator is arranged *» comparing said correlation measure, r. against said further thresh- 
old value TH2, and conditioning said means for producing a block of substitute pixel data to orovSc sak TJS, 
^W^ele^tor^^nH^^i" X < TH^;'' correlation measure, r, is less than TH2, arri to^^wWe a Wo^^l^a^^from 

1 °" i^rS.^ forth in C,a L m 8 Characterized in lt ™t said said apparatus responsive to data representing imaoe 
tSXSZZSZSl ad,aCert ° f ^ ^ date " for a — 'of SEES 

said bfiSE Se d . e dS! inin9 ^ ,6aSt re ' atiVe COrre,ati ° n ° f im39e 9radientS in ^ of ^' data «*"*•« 
11 - iSlt^^T^f SSt forth ln C ' aim 10 ^^^ri^ed in that said said apparatus responsive to data representing imaoe 

K^2E££2T b, °* 01 h - pix * date - for ™* a — ' « ST ? 

of bad and determinin9 *" ^ C ° fre ' ati0n * im39e m ° ti0n ^ * data ad ' ac ^ "ock 

apparatus for providing the weighted sum of values representing said relative correlation of imane motion 
and saKJ relative correlation of image gradients to generate said correlation measure J*" 6 '*' 0 " * ,mage motlon 

ES^^ESSE" -»*»* — da -- * 'eterrrtning a meagre. T^SSSH 

a . a for determining a temporal conelation measure between a block adjacent said block of bad pixel data 

and a block corresponding to a predicted block of a block adjacent said block of bad pixel data and 
said reS^l^ " 9 « ^'^ng relative correlation of image motion 

measuJe^ ,ma96 >nd temp0ra ' COrre,atfon measure - to generate said Correlation 

13. The apparatus set forth in claim 11 characterized in that said said apparatus for determining Ihe relative correlation 
so of .mage motion includes apparatus for calculating the retative correlation of motion vector^ asscSJwX SoSs 
of compressed v.deo data adjacent said block of bad pixel data. associated wrth blocks 

1 * a^£S; 9e ^ 

« P™^'" 0 . of P ixel data and an error signal indicating which of said blocks are bad blocks- 

determining a dominant image gradient in blocks of pixels adjacent a bad block- 

M jsEssttzzsr ** *■*"' *~ oraaeot ' ******* mo< * - ^ ■» 

Bewail, predicting a sttosetute block ol data to. said ted Hoc* In reapers* to said error signal; 
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converting the spatially interpolated substitute block to a block of transform coefficients. TC1 . representing 
frequency spectra; 

converting the temporally predicted substitute block to a block of transform coefficients, TC2, representing 
frequency spectra; 

forming a further block of coefficients by selecting respective coefficients from said blocks of transform coef- 
ficients TC1 and TC2 according to a predetermined criteria; 

inverse transforming said further block of coefficients to produce a substitute block of pixel data. 

1 5. The method set forth in Claim 1 4, characterized by determining a dominant image gradient in blocks of pixels adja- 
cent a bad block, and spatially interpolating in accordance with said dominant image gradient; wherein said step of 
selecting comprises: 

selecting from said block of transform coefficients TC1, only coefficients representing image information 
along the dominant gradient; and 

selecting from the block of transform coefficients TC2, the remaining coefficients. 

16. The method set forth in claim 14 characterized in that said the step of selecting comprises: 

providing an error value "e"; 

determining differences between corresponding coefficients in the blocks of transform coefficients TC1 and 
TC2; and 

if the respective difference is less than the error value "e". selecting the coefficient from the block of trans- 
form coefficients TC1 , else selecting the respective coefficient from the block of transform coefficients TC2. 

17. The method set forth in claim 16 characterized in that said the step of providing the value "e", comprises: 

providing a block of data adjacent the bad block, and a temporally predicted block corresponding to said 
block of data adjacent said bad block; 

determining differences between corresponding data in said temporally predicted block and the correspond- 
ing block; and 

statistically analyzing the differences and generating the value "e" from a statistic of the differences. 
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